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ABSTRACT 
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The current state of the art techniques of risk assessment rely on checklists and 
human expertise. This constitutes a weak approach because different people could arrive 
at different conclusions from the same scenario. The difficulty of estimating the duration 
of projects applying evolutionary software processes adds intricacy to the risk assessment 
problem. This dissertation introduces a formal method to assess the risk and the duration 
of software projects automatically, based on measurements that can be obtained early in 
the development process. The method has been designed according to the characteristics 
of evolutionary software processes, such as efficiency, requirement volatility and 
complexity. The formal model based on these three indicators estimates the duration and 
risk of evolutionary software processes. The approach introduces benefits in two fields: 

a) Automation of risk assessment. 

b) Early estimation methods for evolutionary software processes. 
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I. INTRODUCTION 



A. THE IMMATURITY OF SOFTWARE ENGINEERING 

"Despite 50 years of progress, the software industry remains years-perhaps 
decades-short of the mature engineering discipline needed to meet the demands of an 
information-age society" (Gibbs, 1994). Much research has analyzed this problem using 
various approaches: formal methods, prototyping, software processes, etc. However, 
Gibb’s assertion still remains true today. 

Since the creation of the first computers, tremendous progress has been made in 
terms of hardware. The introduction of the general-purpose computer has been especially 
important because of its versatility. The stored program allowed specialized applications 
created by software. These applications have grown in size and complexity covering 
numerous human activities. Unfortunately, the ability to build software has not followed 
the same rate of progress (Hall, 1997. pp xv). Gerald Weinberg said, "to call software 
development an infant discipline is not a moral judgment, but merely a colorful way to 
summarize its short history and present existence.". (Gilb, 1977. Foreword). Software 
engineering focuses on planning, developing and maintaining software products. Clearly, 
the creation of software imposes different challenges from the creation of hardware. 

Experience suggests that building and integrating software by mechanically 
processable formal models leads to cheaper, faster and more reliable products (Luqi, 
1997). Software development processes, such as the Hypergraph model for software 
evolution (Luqi, 1997) and the Spiral model (Boehm, 1988) have improved the state of 
the art. However, they share a common weakness: risk assessment. This dissertation 
addresses this issue. 
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In the software evolution domain, risk assessment has not been addressed as part 
of the previous models. In the various enhancements and extensions, the graph model 
does not include risk assessment steps, hence risk management remains a human- 
dependent activity that requires expertise. This dissertation provides a formal model to 
assess the project risk independently of the project manager’s experience. 

On the evaluation of the spiral model, one of the difficulties mentioned by Boehm 
was: "Relying on risk-assessment expertise, the spiral model places a great deal of 
reliance on the ability of software developers to identify and manage sources of project 
risk. . . Another concern is that a risk-driven specification will also be people-dependent." 
(Boehm, 1988). 

Why has software engineering not reached the maturity level of other forms of 
engineering? Perhaps the answer lies in the differences between software engineering and 
other disciplines. One difference is that software engineering is highly dependent on 
people. A second difference is that software engineering is younger (only forty years 
versus centuries for civil engineering). A third difference is that the product, software, is 
intangible. Estimating software’s real value at the beginning of the software process is 
difficult. All these differences create a great deal of uncertainty and equivocality 1 in 
software development projects. 

Many investigations (Boehm, 1989), (Charette, 1997), (Gilb, 1988), (Hall, 1997), 
(Jones, 1994), (Karolak, 1996), (SEI, 1996) have addressed the problem of risk 
assessment following the perspective of the traditional disciplines. The tools for risk 
assessment are guidelines for practices, checklists, taxonomies of risk factors and few 
metrics. All these methods work fine if there is a project manager trained in risk 
assessment with enough experience. Such personnel are very scarce. That is one of the 
reasons why software engineering is still immature. Software costs are often misestimated 

1 The term equivocality introduced by (Burton & Obel, 1998) means the amount of ignorance about the 
variables in a system. 
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and many projects exceed their schedules and budgets. This dissertation provides a formal 
way to assess the risk of a software project independently of the experience of the 
decision-maker. 



B. RISK AND THE ESTIMATION PROBLEM 

As the range and complexity of computer applications have grown, the cost of 
software development has become the major expense of computer-based systems 
(Boehm, 1981), (Karolak, 1996). Research shows that in private industry as well as in 
government environments, schedule and cost overruns are tragically common (Luqi, 
1989), (Jones, 1994), (Boehm, 1981). Developing software is still a high-risk activity. 
Research shows that 45 percent of all the causes for delayed software deliveries are 
related to organizational issues (vanGenuchten, 1991). Despite the advances in 
technology and CASE tools, little progress has been done to improve the management of 
software development projects (Hall, 1997). The acquisition and development 
communities, both governmental and industrial, lack systematic ways of identifying, 
communicating and resolving technical uncertainty (SEI, 1996). 

This research focuses on software project risk assessment, namely predicting the 
success of the project. The only ways to evaluate the degree of success of a project are a) 
to compare planned and actual schedules; b) to compare planned and actual costs; and c) 
to compare planned and actual product characteristics. Software reliability, an emergent 
branch of software engineering, has addressed this last part. However, the first two issues 
have not yet been emphatically addressed. 

For many years research has greatly increased our knowledge of software projects. 
Among such software laws, it is known that: 

• Manpower and time are not interchangeable (Brooks, 1974). 

• Human productivity rates are highly variable, and function and size are 
highly correlated with errors and duration of the project (Putnam, 1980). 
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• The majority and most costly errors are introduced during the requirements 
phase (Boehm 1981). 

• Life-cycle manpower patterns follow tailed probability curves (Norden, 
1963), (Putnam, 1980, 1992, 1996, 1997), (Boehm, 1981). 

• Standards, good practices, guidelines, and heuristics improve the 
development process (Humphrey, 1989). 

Now CASE tools that improve the productivity exist. Macro models also can 
estimate with different degrees of success the effort and duration of software projects 
(Albrecht, 1979), (Boehm, 1981, 2000), (Putnam, 1997). What is not available is a model 
of the internal phenomenology of the software life cycle. Without the knowledge of such 
a model, scientific risk assessment is almost impossible. This dissertation provides a 
model to explain the risk of the projects. 

In this dissertation, risk is defined as the product of a future outcome times the 
probability of an occurrence of such an outcome. The outcome could be negative, a loss 
(this is the general approach that all previous research has applied), but also positive 
leading to an opportunity. 



C. RESEARCH QUESTIONS 

The software process is a set of activities with dependency relationships that occur 
over a certain period of time. From this point of view software projects do not differ from 
any other type of project. The difference is that for software, the number of activities is 
uncertain until late in the development process. At the beginning of such a process, a 
great deal of uncertainty exists. This uncertainty can be reduced through effort, which can 
be expressed in terms of time and cost. As time goes by, the level of uncertainty usually 
decreases because more information becomes available. Unfortunately, the main 
resources (time and budget) also exhibit the same behavior. So project managers, as 
decision-makers, must choose between making early decisions with a great deal of 
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uncertainty, or postponing decisions by trading time for information. This leads to the 
basic research question addressed in this dissertation: 

What are the automatically collectable early measures of the software 

process that indicate project risk? 

The concept of early measure is emphasized because recognizing the risks in the 
early phases increases the probability of success, improving, consequently, the 
competitive advantage. This research focuses on automatically collectable measures 
because risk identification should not impose a significant extra workload and must be as 
objective as possible. This leads to the second question: 

How can these measures be used to assess project risk? 



D. GENERAL APPROACH 

Despite the recent improvements in software processes and automated tools, risk 
assessment for software projects remains an unstructured problem dependent on human 
expertise (Boehm, 1988), (Hall, 1997). This dissertation explores ways to transform risk 
assessment into a structured problem with systematic solutions. Solving the risk 
assessment problem with indicators measured in the early phases would constitute a great 
benefit to software engineering. In the requirements phase, changes can be made with the 
least impact on the budget and schedule. The requirements phase is the crucial stage to 
assess risk because: a) it has a huge amount of human intervention and communication 
that can be misunderstood and can be a source of errors; b) errors introduced at this phase 
are very expensive to correct if they are discovered late; c) the existence of generation 
tools diminishes the errors in the development process if the requirements are correct; and 
d) requirements evolve introducing changes and maintenance along the whole life cycle. 

Constructing a model to assess risk based on objective measurable parameters that 
can be automatically collected and analyzed is necessary. One of the goals of this research 
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is to integrate a risk assessment model with the previous research (Luqi, 1988) on CAPS 2 
at NPS. This integration is required in order to capture metrics automatically and to 
provide project managers with a more complete tool. 

Software risk management includes identifying, assessing and mitigating risks. It 
requires dealing with complexity and assigning scarce resources in the most efficient way. 
The scope of this dissertation is limited to risk identification and risk assessment. 
Automated methods can provide major impact in these two areas. 

This dissertation studies project risk assessment by dividing it into three classes: 
resource risk assessment, process risk assessment, and product risk assessment. A 
dependency between these classes of risk exists. The success of the project depends on 
the matching between the characteristics of the process, the resources, and the product. 

A measure of project success is the probability (p) of developing the required 
product according to the planned schedule and within the budget applying a certain 
software process. Consequently, the project risk is the cost associated with its failure 
times the probability of failure (q = 1 - p). The hypothesis of this dissertation is that the 
associated probability distribution is skewed to the right. 

Creating a set of metrics customized to the characteristics of software evolution 
including complexity, requirements volatility and efficiency is necessary to support the 
estimation of risk. The details of such a framework are described in Chapter IH. The 
approach has a fundamental implication: in order to assess risk, one must assess duration 
of the project and consequently the effort (see Chapter II Section E). 



2 CAPS stands for Computer Aided Prototyping System 
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E. SOFTWARE EVOLUTION FOCUS 

Boehm has shown that early parts of the system development cycle, such as 
requirements and design specifications, are especially prone to errors (Boehm, 1981). As 
a result, problems originating in the early stages often have a lasting influence on the 
reliability, safety and cost of the system. Evolutionary prototyping offers an iterative 
approach to requirements engineering to alleviate the problems of uncertainty, ambiguity 
and inconsistency inherent in the process. Moreover, prototyping can improve the capture 
of change in requirements and assumptions during the development process. This effect is 
particularly observed in projects involving multiple stakeholders with different points of 
view (Ramesh, 1995), (Conklin, 1988). Software prototyping is characterized by changes 
in the requirements. Such changes are consequence of the evolution and refinements. For 
that reason the requirements volatility is one of the metrics used in our model. 

Evolutionary driven computer aided software engineering (CASE) tools for 
computer-aided prototyping provide a logical assessment of the consistency and clarity of 
requirements and specifications. The use of prototypes facilitates the requirement phase 
in any type of software projects. Particularly, in real-time applications where severe time 
constraints impose more challenges, the use of prototypes helps to describe the 
requirements in a clear, precise, consistent and executable format. Prototypes can be 
applied to demonstrate system scenarios to the affected parties as a way: a) to collect 
criticisms and feedback that are sources for new requirements; b) to detect deviations 
from users’ expectations early; c) to trace the evolution of the requirements; d) to improve 
the communication and integration of the users and the development personnel; and e) to 
provide early warning of mismatches between proposed software architectures and the 
conceptual structure of requirements. 

The benefits of prototyping are unquestionable. All modem life-cycle models, 
such as Bohem’s spiral, Luqi’s graph model, rapid application development (RAD), etc., 
are based on prototyping. Experience suggests that building and integrating software by 
mechanically processable formal models lead to cheaper, earlier and more reliable 
products (Luqi, 1997). 
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Despite their benefits, evolutionary software processes have two issues that must 
be solved. The first concern is that they rely on human expertise to identify and assess 
risk. This dissertation addresses this issue in Chapter VII. 

A second concern in the use of prototypes is that they introduce a problem in 
project planning because of the uncertain number of prototyping cycles required before 
constructing the product and the amount of complexity that should be covered at each 
cycle. For the most part, existing project management and estimation techniques are 
based on linear layouts of activities. CPM and PERT techniques are not well-suited to 
deal with cycles because they are based on acyclic digraphs. This issue is discussed in 
Chapter II Section G. 



F. CONTRIBUTIONS 

The first contribution of this dissertation is the transformation of the unstructured 
problem of software risk assessment into a structured one (Chapter HO. This contribution 
impacts the software engineering state of the art, but also risk management in general. 
The use of formal models based on a set of metrics solves the human -dependency issue 
characteristic the present state of the art in that area. The set of metrics chosen includes 
the principal characteristics of any evolutionary software process, although it could be 
expanded to other indicators in future research, includes the principal characteristics of 
any evolutionary software process. 

A second contribution, and perhaps the most important, is the creation of 
estimation models that can be used from the beginning of the project (Chapter V, Section 
C; Chapter VI). These models address the requirement issue of the present state of the art 
estimation models, which rely on an unambiguous and frozen definition of requirements. 
With the proposed models, project managers will have a decision support tool much 
earlier in the life cycle. The dissertation shows that commonly used planning techniques, 
such as Pert, Gantt, and CPM, could result in overly optimistic results when they are 
applied to communication-intensive projects like software development. 
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The third contribution is related to software metrics. The risk assessment 
framework and the estimation models rely on measures that are innovations in the field of 
software metrics: 

• Two complexity metrics specially suited for formal specifications (Chapter 
V, Section A. 3). 

• An organizational productivity metric (Chapter V, Section A.2). 

• A requirement volatility metric that constitutes by itself a decision support 
tool (Chapter V, Section A.l). 

A fourth contribution addresses the lack of risk assessment in the evolutionary 
software processes. This dissertation improves the Relational Hypergraph Model by the 
introduction of a new step addressing risk (Chapter YD, Section A). 



G. ORGANIZATION OF DISSERTATION 

This dissertation is organized in nine chapters. The introduction is in the present 
chapter. Chapter 13 presents relevant theoretical foundations and background on software 
engineering, software evolution, organizational theory, chaos theory, estimation models, 
software reliability, and risk management. The conceptual framework of the model is 
developed in Chapter E3. Chapter IV presents the detailed research design. The 
development of the model, the statistical analysis of the observed results, and the 
algorithms for estimation are presented in Chapters V and VI. Chapter VII discusses 
integration with CAPS, introduces an improvement to the evolutionary software process, 
and discusses the method for risk assessment. Finally, Chapter VIH presents the 
conclusions and identifies opportunities for future research. 



9 



THIS PAGE INTENTIONALLY LEFT BLANK 



10 



II. THEORETICAL FOUNDATION 



A. THEORETICAL FOUNDATION FOR SOFTWARE EVOLUTION 

1. The Graph Model 

The graph model is a data-graph model for evolution that records dependencies 
and supports automatic project planning, scheduling, and configuration management. The 
evolution process is represented by a graph that at any given moment models the current, 
past, and planned future states of the software system. 

The graph model has experienced its own evolution process. Luqi introduced the 
original simple version of the model (Luqi, 1989). Mostov and Luqi refined and 
elaborated the model (Mostov, 1989, 1990), (Luqi, 1990). Luqi introduced the notion of 
hypergraph to realize automated software evolution in multidimensional phases (Luqi, 
1990). Further refinements, including scheduling and team coordination, were introduced 
by (Badr, 1993). Conflict resolution of requirements and criticisms were introduced by 
(Ramesh, 1992) and (Ibrahim, 1996). Luqi extended the graph model to a hypergraph 
that improved the traceability of dependencies and introduced the concept of hyper- 
requirements (Luqi, 1997). Finally, Ham extended the model to a relational hypergraph 
model (Ham, 1998a, Ham, 1998b, Ham, 1998c). 

2. Conflict Resolution Model 

Evolutionary software development requires a way to resolve the conflicts that 
could occur between various users’ points of view. System design must follow a 
deliberation process that involves resolving issues that should be addressed to satisfy user 
requirements. Conklin (Conklin, 1988) applied the IBIS model to resolve conflicts in 
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requirements. Later Ramesh (Ramesh, 1992) 
introduced the REMAP model that relates the 
following concepts: 

(1) Requirements represent the goals to be 
satisfied by the design process. 

(2) Issues are questions or concerns that 
different stakeholders introduce. 

(3) Positions are alternatives that address an 
issue. 

(4) Arguments either support or object a 
position. 

(5) Decisions represent the resolution of issues 
and lead to constraints. 

This approach seems to have inspired the Win Win at USC. Win Win is a 
methodology that aids in the capture, negotiation, and coordination of requirements for 
large systems. It assumes that a group of people, called stakeholders, has different, 
probably contradictory, views and positions that require being harmonized in order to 
elucidate requirements (USC, 2000). Win Win has been implemented for Solaris, SunOS, 
Linux, and Java. 




Figure 2.1: REMAP model 



3. Relational Hypergraph Model 

The relational hypergraph model, introduced in (Ham99e), is a formal model for 
software evolution that incorporates the features of the previous graph models. The 
hypergraph model (Luqi, 1997) represents the evolution history, as well as the plan for 
the future, a hypergraph. A hypergraph is a directed graph with hyperedges, which may 
have multiple input and output nodes. The formal definition of the relational hypergraph 
model is based on the following definitions: 
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Definition 1: Directed hypergraph (Ham, 1999f). A directed hypergraph is a tuple 
H = (N, E, I, O) where N is a set of nodes, E is a set of hyperedges, I is a function giving 
the set of input nodes of each hyperedge, and O is a function giving the output nodes of 
each hyperedge. 

Definition 2: Path (Ham, 1999f). A path p from node ni to node n k is a sequence 
of hyperedges ei, ..., e k .i (k>0), and a sequence of nodes ni, ..., n k , such that 
n; s I(eO and nj+i € O(ej) for 1 < i < k. 



Definition 3: Acyclic hypergraph (Ham, 1999f). A hypergraph H = (N, E, I, O) is 
acyclic if and only if there is no path from any node in H to itself. 

Definition 4: Reachable (Ham, 1999f). A set N of nodes is reachable from a set R 
of nodes if and only if there is a path to each node n € N from some node re R. A 
hypergraph H, is reachable from a set R of its nodes, if and only if all its nodes are 
reachable from R. The root of the hypergraph H is a node from which H is reachable. A 
leaf of H is a node from which no other node is reachable. 

Definition 5: Composite node and composite edge (Ham, 1999f). A composite 
node is a set of nodes, and a composite edge is a set of edges. 

Definition 6: Hypergraph set (Ham, 1999f). A hypergraph set is the union of 
nodes and edges of a set of hypergraphs. 

Definition 7: Refinement of a composite node (Ham, 1999f). Let H = (N, E, I, O). 
The refinement of a composite node n e N is a directed minimal hypergraph H m = (Nj n u 
N 0 ut> {e}, I, O), where the input node set Nj„ = {ni, n„}, the output node set N ou t = { n } , 
and the edge set is {e}. The edge e is called a decomposition edge and relates the node to 
the nodes in its decomposition. 
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Definition 8: Opposite hypergraph (Ham, 1999f). Let H = (N, E, I, O) then its 
opposite hypergraph is H op = (N, E, O, I). 

Definition 9: Hyperpath (Ham, 1999f). A hyperpath from Nj to N 2 in the 
hypergraph H = (N, E, I, O), is a minimal hypergraph from a set of nodes Ni to another 
set of nodes N 2 where Nj c N and N 2 c N. 

Definition 10: Refinement of a composite edge (Ham, 1999f). Let H = (N, E, I, 
O). The refinement of a composite edge e = {ei, ..., e n , m, ..., n n ), where is a hypergraph 
set of minimal hypergraphs R = (Ni„ u N 0 U t, e, I, O). Ni n = 1(e), N ou t = O(e), and e 1 , ..., e n 
are called subedges. 

Definition 11: Refinement of a minimal hypergraph (Ham, 1999f). Let H m = (Nj n 
u N out , {e}, I, O) be a minimal hypergraph. The refinement of a minimal hypergraph is a 
hypergraph set R = Hj n u H ou , u H e , where Hj n is a refinement of Nj n , H ou t is a refinement 
of N ou t, and H e is a refinement of e. H m can be viewed as a graph composed of two nodes 
(Nin, N 0 ut) and one edge (e) where Nj n and N 0U t are hypergraphs and e is hyperedge. 

Definition 12: Evolutionary hypergraph (Ham, 1999f). An evolutionary 
hypergraph is a labeled, directed, and acyclic hypergraph H = (N, E, I, O) together with 
label functions that give component attributes to the nodes and step attributes to the 
edges. 



Definition 13: Top-level evolution step (Ham, 1999f). A hyperedge is called a 
top-level evolution step if there are no parent evolution steps. 

Definition 14: Atomic evolution step (Ham, 1999f). An atomic evolution step is 
an atomic edge without any refinements. 
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Definition 15: Top-level evolutionary hypergraph (Ham, 1999f). A top-level 
evolutionary hypergraph is an evolutionary hypergraph whose edges are top-level 
evolution steps. 

Definition 16: Atomic evolutionary hypergraph (Ham, \999f). An atomic 
evolutionary hypergraph is an evolutionary hypergraph with atomic evolution steps as its 
hyperedge. 

Definition 17: Primary input (Ham, 1999f). Primary inputs are previous versions 
of the output component of an evolutionary step. 

Definition 18: Secondary inputs (Ham, 1999f). Secondary inputs are all input 
components required in an evolutionary step that are not primary inputs. 

Definition 19: Primary-input-driven hypergraph (Ham, 1999f). An evolutionary 
hypergraph is called primary-input-driven if and only if its input nodes are primary inputs. 

Definition 20: Secondary-input-driven hypergraph (Ham, 1999f). An evolutionary 
hypergraph is called secondary-input-driven if and only if its input nodes are secondary 
inputs. 



Definition 21: Relational hypergraph (Ham, 1999f). A relational hypergraph is an 
evolutionary hypergraph in which the dependency relationships between components and 
steps can have a hierarchy of specialized interpretations. 

Definition 22: Software prototyping demo step (Ham, 1999f). A software 
prototyping demo step is a step in which the input components are a set of criticisms 
(Cl), a set of programs (P), a set test scenarios (T), and a set of stakeholders (U), 
producing an output component set of criticisms (C2). 
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Definition 23: Issue analysis step (Ham, 1999f). A issue analysis step is a step in 
which the input components are a set of previous issues (J 1 ), a set of stakeholders (U), a 
set of criticisms (C), producing an output component set of new issues (J2). 

Definition 24: Requirement analysis step (Ham, 1999f). A requirement analysis 
step is a step in which the input components are a set of previous requirements (Rl), a set 
of issues (J), a set of stakeholders (U), producing an output component set of new 
requirements (R2). 

Definition 25: Specification design step (Ham, 1999f). A specification design step 
is a step in which the input components are a set of previous specifications (SI), a set of 
stakeholders (U), a set of requirements (R), producing an output component set of new 
specifications (S2). 

Definition 26: Module implementation step (Ham, 1999f). A module 
implementation step is a step in which the input components are a set of previous 
modules (Ml), a set of stakeholders (U), a set of specifications (S), producing an output 
component set of new modules (M2). 

Definition 27: Program integration step (Ham, 1999f). A program integration step 
is a step in which the input components are a set pf previous programs (PI), a set of 
stakeholders (U), a set of modules (M), producing an output component set of new 
programs (P2). 

Definition 28: Software product demo step (Ham, 1999f). A software product 
demo step is a step in which the input components are a set of previous optimizations 
(Kl), a set of stakeholders (U), a set of programs (P), a set of test scenarios (T), 
producing an output component set of new optimizations (K2). 



16 



Definition 29: Software product implementation step (Ham, 1999f). A software 
product implementation step is a step in which the input components are a set of previous 
versions of programs (PI), a set of stakeholders (U), a set of optimizations (K), producing 
an output component set of new programs (P2). 

Definition 30: Software prototyping evolution process (Ham, 1999f). A software 
prototyping evolution step is a hypergraph with a path with the following properties: 

• Steps are software prototype or product demo, issue analysis, requirement 
analysis, specification design, module implementation and program 
integration. 

• Nodes are old version programs, criticisms, issues, requirements, 
specifications, modules, and new version programs. 

Definition 31: Software product generation process (Ham, 1999T). A software 
product process is a relational hypergraph with a path with the following properties: 

• Steps are software prototype or product demo, and program integration. 

• Nodes are new version prototypes or old version programs, optimizations, 
and new version programs. 

Definition 32: Software evolution process (Ham, 1999f). A software evolution 
process is a relational hypergraph with a combined structure of software prototyping 
evolution processes and software product generation processes. 

Definition 33 3 : Top-level relational hypergraph net (Ham, 1999f). A top-level 
relational hypergraph is a set composed from a set of primary inputs, one or more sets of 
secondary inputs, and a set of output nodes of a top-level evolution step. (Ham, 1999f) 
called this is concept SPIDER (Step Processed in Different Entrance Relationships). 



3 This definition is presented for illustration purposes and completeness, but will not be addressed in this 
dissertation. 
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Definition 34 4 : Atomic relational hypergraph net (Ham, 1999f). An atomic 
relational hypergraph is a set composed by a set of primary inputs, one or more sets of 
secondary inputs, and a set of output nodes to an atomic evolution step. (Ham, 1999f) 
called this concept an atomic SPIDER. 



4. Computer Aided Prototyping System (CAPS) 

Computer Aided Prototyping System (CAPS) is a CASE tool that provides a 
collection of techniques and languages for computer-aided prototyping, including logical 
assessment of the consistency and clarity of requirements and specifications. CAPS 
methods involve the use of real-time constraints and abstract modeling to describe the 
requirements in a clear, precise, consistent and executable format. Prototypes can be 
applied to demonstrate system scenarios to the affected parties as a way to elucidate 
requirements. 

Real-time systems present special difficulties in terms of requirement engineering. 
Some requirements are difficult to provide for the user, and difficult to determine for the 
analysts. The best way to discover these hidden requirements is via prototyping. CAPS is 
a tool specially suited for this task. It has a graphical, easy to understand, interface that 
maps to a specification language, which in turns generates Ada code. The main 
components of CAPS are: 

(a) The prototype system description language (PSDL). 

(b) User interface based on a graphic editor with a palette of objects that include 
operators, inputs, outputs, data flows and operator loops. 

(c) The software database system provides a repository for reusable PSDL 
components. A search engine helps the designer to find reusable components. 

(d) The execution support system consists of a translator, scheduling mechanisms, 
execution monitors, and a debugger. 

4 This definition is presented for illustration purposes and completeness, but will not be addressed in this 
dissertation. 
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The prototyping process consists of prototype construction and modification 
(evolution) based on evolving requirements and code generation. Both construction and 
modification are exploratory activities with a common target: to satisfy multiple users 
with different and often conflicting points of view. Requirement engineering is a 
consensus-driven activity in which mechanisms for conflict resolution and traceability of 
requirement evolution represent critical success factors. 

PSDL is based on data flow under real-time constraints and uses an enhanced data 
flow diagram that includes non-procedural control and timing constraints. PSDL serves as 
an executable prototyping language at a specification or design level. The user interface 
contains a graphic editor, a browser to view reusable components, and an expert system 
that provides the capability to generate English text descriptions of PSDL specifications. 

The software database system provides the repository facilities for reusable 
components, and for control of versions. The execution support system consists of a 
translator that generates code that binds the reusable components, scheduling 
mechanisms, and a debugger. 

The model views a software evolution process as a partially ordered set of steps. 
Steps represent activities required to produce the system. A step has states that reflect the 
dynamic progression of the activity from the moment the step is proposed to the moment 
it is completed or abandoned. 



5. Conclusions about the Relational Hypergraph Model and CAPS 

The precedent definitions constitute the formal specification of the relational 
hypergraph model. They constitute a framework to support the software evolution 
processes. CAPS is based in part on these definitions. The relational hypergraph model is 
supposed to support any software development process, including processes with risk 
assessment steps. However, the model has not been previously applied to such a process. 
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and has not been specialized to include features needed just for risk assessment. This 
issue creates a human dependency in risk assessment. Despite this limitation, the model 
can be extended to support automated risk assessment. Solving this issue is one of the 
goals of this dissertation (see Chapter VET). 



B. THEORETICAL FOUNDATION FOR RISK MANAGEMENT 

The research on risk and risk management is very extensive. It comprises 
operational research, project management, software engineering, and software reliability. 
Operational research provides the theoretical foundation to describe and analyze risk. 
Project management, software engineering, and software reliability apply the theory. This 
research narrows the problem to software, specifically to the software engineering 
domain. 



1. Risk and Uncertainty 

Developing software is still a high-risk activity. Despite the advances in 
technology and CASE tools, little has improved the management of software 
development projects. The acquisition and development communities, both governmental 
and industrial, lack a systematic way to identify, communicate and resolve technical 
uncertainties (SEI, 1996). Research shows that 45 percent of all delayed software 
deliveries are related to organization issues (vanGenuchten, 1991). Software is the main 
expense in computer systems (Boehm, 1981), (Karolak, 1996). Besides the improvements 
in tools and methodologies, there is little evidence of success in improving the process of 
moving from the concept to the product. A study published by the Stadish Group reveals 
that the number of software projects that fail has dropped from 40% in 1997 to 26% in 
1999. However, the percentage of projects with cost and schedule overruns rose from 
33% in 1997 to 46% in 1999 (Reel, 1999). 
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Part of the problem is misinterpreting the importance of risk management. It is 
usually and incorrectly viewed as an additional activity layered on the assigned work, or 
worse, as an outside activity that is not part of the software process (Hall, 1997), 
(Karolak, 1996). 

A second source of problems in risk management is the lack of tools (Karolak, 
1996). The main reason for this lack of tools is that risk assessment is apparently an 
unstructured problem. Structured problems involve routine and repetitive problems for 
which a standard solution exists. Unstructured problems require decision-making based 
on a three-phase method (intelligence, design, choice) (Turban & Aronson, 1998). An 
unstructured problem is one in which none of the three phases is structured. 

Risk management is highly biased by a manager’s perceptions and characteristics, 
which are difficult to represent by an algorithm. Depending on the decision-maker’s risk 
behavior, he or she can decide early with little information, or can postpone the decision, 
gaining time to obtain more information, but losing some control. 

A third source of the risk management problem is the confusion created by the 
informal use of terms. Often, the software engineering community (and most parts of the 
project management community (Wideman, 1992)) use the term "risk" casually. This 
term is often used to describe different concepts. It is erroneously used as a synonym of 
"uncertainty" and "threat" (SEI, 1996), (Hall, 1997), (Karolak, 1996). 

In this research the term "risk" is reserved to indicate the probabilistic outcome of 
a succession of states of nature, and the term "threat" is used to identify the dangers that 
can occur. Generally, software risk is viewed as a measure of the likelihood of an 
unsatisfactory outcome and an expected loss affecting the software from different points 
of view: project, process, and product (Hall, 1997), (SEI, 1996). However, this definition 
of risk is misleading because it confounds the concepts of risk and uncertainty. In general, 
most parts of the decision-making in software processes is under uncertainty rather than 
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under risk. The definition of risk presented in Chapter I stated that risk is the product of 
the value of an outcome times its probability of occurrence. This outcome could be 
positive (gain) or negative (loss). This abstraction permits one to address not only the 
classical risk management issue, but also to discover opportunities leading to competitive 
advantage. Now let’s discuss briefly the decision-making environments in order to clarify 
these concepts. 

Three possible situations exist in any decision context: certainty, risk and 
uncertainty. Decisions under "certainty" occur when the decision-maker knows the exact 
consequence of each alternative or decision. In this case the decision process is very 
simple: the alternative with the best outcome is chosen. However, this is a rarity. 

Usually the decision-maker does not have a complete picture of the future, but 
knows the probability of occurrences of the various possible states of nature. In this case 
the decision-making is under risk, and many techniques can be addressed to support the 
decision: expected monetary value, expected value of perfect information, opportunity 
loss, and sensitivity analysis, among others (Render, 1997). All these methods rely on the 
huge hypothesis of knowing the exact probability for each state of nature. 

A completely different situation is when the decision-maker does not have 
precise information about the probability distribution of the different states of nature. In 
these cases a completely different set of techniques must be applied to support the 
decision-making process: maximin, minimax, Laplace, Hurwicz, or minimax regret 
(Render, 1997). These methods are explained in Section B.2. 

The distinction between risk and uncertainty is important for decision making 
because it leads to drastically different approaches to risk assessment: 

(a) Assessing Software Risk by Measuring Reliability. In this case the decision- 
making is under risk. However, uncertainty exists even using probabilistic models. 
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This is created by uncertainties in parameter values, uncertainties in modeling, and 
ambiguities in the degree of completeness such as: (Baybutt, 1989) 

• Ambiguities in parameter values are consequences of the need to estimate 
parameter values from data. These ambiguities arise because the available data is 
usually incomplete and because the analyst, depending on incomplete knowledge, 
makes inferences. 

• Deficiencies of the model in representing reality. 

• Completeness ambiguities are introduced by the analyst’s inability to evaluate 
exhaustively all contributions to risk. 

(b) Assessing Software Risk Using Practices and Guidelines (SEI, 1996). In this case 
there is no probabilistic model to rely on, hence the decision-making is under 
uncertainty. 

It follows, as previously stated, that most software-managers’ decisions are made 
under uncertainty. Three groups of researchers view the issue from different angles. The 
researchers who follow the probabilistic approach have successfully assessed the 
reliability of the product (Lyu, 1995), (Schneidewind, 1975), (Musa, 1998). However, this 
approach assesses software reliability when it is too late to economically correct possible 
faults because the product is complete or almost complete. This approach is discussed in 
Section E. 

Other researchers assess the risk from the beginning, in parallel with the 
development process. However, in this case, the approach is less rigorous and 
unstructured. Basically the proposals are lists of practices and checklists (SEI, 1996), 
(Hall, 1997) or scoring techniques (Karolak, 1996). Paradoxically, SEI defines software 
technical risk as a measure of the probability and severity of adverse effects in developing 
software that does not meet its intended functions and performance requirements (SEI, 
1996). However, the term "probability" in this case is misleading because the applicable 
probability distribution is unknown. This approach is discussed in Section C. 



23 



(c) Assessing Software Risk by Using Estimation Models. A third group of researchers 
focus mainly on the estimation of effort and time that has characteristics of both 
previous groups. This approach is tangentially related to risk and will be discussed in 
Section D. 



2. Decision under Uncertainty 

Quite frequently decision-makers must use incomplete information. Particularly, 
the problem of decision-making under uncertainty involves choosing among a set of 
alternatives under the following conditions: 

• The outcome of each course of action depends on several possible states of 
nature. 

• The outcome for each alternative under each state of nature is known. 

• The probability of occurrence of each state of nature is unknown or known 
only very roughly. 

When the probability of occurrence of each state of nature is unknown or cannot 
be assessed, then the following five techniques can be applied: 

(1) Maximax Criterion . This criterion implies an optimistic vision of the future. The 
method consists of choosing the alternative that maximizes the outcome for every 
state of nature. 

(2) Maximin Criterion . This method finds the alternative that maximizes the 
minimum outcome. It represents a pessimistic approach. 

(3) Laplace Criterion . This method uses equal probabilities for each state of nature 
and then computes the outcomes for each state of nature, choosing the highest 
outcome. 

(4) Criterion of_ Realism . This method is also known as Hurwicz Criterion (Render, 
1997). It is a compromise between an optimistic and a pessimistic decision. The 
decision-maker must choose a coefficient of realism a between 0 and 1. This 
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coefficient is applied to the most favorable state of nature outcome, and (1 - a) is 
applied to the outcome of the most unfavorable state of nature. The alternative 
with the higher weighted sum is chosen. 

(5) Minimax Cnterion . This method is based on opportunity loss. The opportunity 
loss refers to the loss that would be suffered by making the wrong decision 
(Render, 1997). The method finds the alternative that minimizes the maximum 
opportunity loss within the alternatives. 

There is no objective way to decide which of these approaches is the best. The 
final result comes from the comparison between the decision made and the reality. The 
choice between them relies on the preferences of the decision-maker and his acceptance 
or avoidance of risk. 



3. Subjective Probabilities and Utility Theory 

Another way to deal with uncertainty situations is to subjectively estimate the 
probabilities of occurrence of the different states of nature. This approach is easy to 
implement but requires a great deal of experience to judge the success probability of each 
alternative. Group consensus techniques (like the Delphi Method) are usually quite 
helpful in such situations (Marshall, 1995), (Putnam, 1992). 

When a decision is made under risk, that is when the probability distribution 
function of the states of nature is known, it is possible to support the decision process 
using decision trees. In general, decision trees based on the expected monetary value 
(EMV) could only lead to bad decisions in many cases. There are many situations in 
which a linear payoff function is unable to represent the behavior of people (Marshall, 
1995). These are the two reasons to study utility theory. In practice, historical data can be 
analyzed to obtain an objective estimate of the outcomes. But in situations, especially 
those that incorporate management decisions, historical data could be irrelevant. The 
judgments and beliefs of the decision-makers may be more important that estimating 
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relevant probabilities (Marshall, 1995). Before describing utility theory in detail, two 
definitions are required: 

The indifference probability for a decision problem between a risky 
venture and a riskless alternative with given known results is that 
probability of success in the risky venture for which the decision-maker is 
indifferent to the two alternatives. (Marshall, 1995). 

The certainty equivalent to a risky venture is the least amount the decision- 
maker would have to obtain for certain by choosing the riskless 
alternative. (Marshall, 1995). 

In many situations the indifference probability and the certainty equivalent would 
have different values for different people. The differences reflect various behaviors 
toward risk. Utility assessment assigns a utility of 0 to the worst outcome and a utility of 
1 to the best outcome. All other outcomes have a utility value between 0 and 1 . When two 
or more alternatives are equally attractive (or unattractive), that is the decision-maker is 
indifferent, then their utility value should be the same. The problem is to find the 
probability that makes the decision-maker indifferent. 

Until now, only one-attribute decision-making problems have been considered. A 
more general scenario would have many attributes for measuring the decision. Often, 
these attributes conflict with each other, hence optimizing one attribute results in 
suboptimizing others. Thus, using trade-offs to resolve such conflicts is necessary. A 
common approach to solving multiattribute problems is to combine the different 
measures into a single numeric measure. The problem can then be treated as single 
attribute problem (Marshall, 1995). In many decision problems, establishing 
measurement criteria is highly challenging, particularly when the decisions are not at the 
operational level. At the operational level, decisions can be measured in terms of lines of 
code or function points. However, at the project management level, the effectiveness of a 
decision could be measured in terms of quality, stability, marketing impact, etc. In such 
cases, multiattribute utility theory should be applied (Fig. 2.2). 
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A1 



OUTCOMES 

A2 



An 




The decision-maker must provide his estimation of return for each attribute 
related to the decision, as a vector R = (Rl, R2, Rn). The decision-maker must also 
introduce his preferences as a weight vector W = (Wl, W2, Wn). The outcomes of 
each attribute are given by Ai, such: 

Ai = Wi * Ri 

n 

where Wi = 1 

i = 0 

The outcome for each alternative is then calculated as a function of the sum of the 
attributes (AI, A2, An) converted to a value between 0 and 1, where 1 is given to the 
best outcome and 0 to the worst. 
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c. 



SOFTWARE ENGINEERING FOUNDATIONS 



1. Software Engineering Institute (SEI) 

The Software Engineering Institute (SEI), at Carnegie Mellon, relies on improving 
the process as a way to improve the products and to diminish risks. This philosophy is 
particularly clear in a guideline created at the request of the USAF by the SEI and Mitre 
Corporation (Humphrey, 1987). The document describes a method to assess the software 
engineering capabilities of contractors. The guideline stated that the quality of the product 
depends on the quality of the process, which in tum depends on the technology used to 
support it, which depends on the maturity level of the organization. Hence, by transitivity, 
the quality of the product depends on the maturity level of the organization. 
Consequently, by assessing the maturity of the organization, one can estimate the 
attributes of the product. SEI proposes a three-dimensional vision of the risk management 
process (SEI, 1996). This vision is as follows: 

(a) The temporal dimension that includes the micro perspective, that is from 
the point of view of the project, and the macro perspective, which covers 
the complete life cycle. 

(b) The methodological dimension that includes practices (software risk 
evaluation (SRE), continuous risk management (CRM) and team-risk 
management (TRM)), and basic constructs including the SETs risk 
taxonomy. 

(c) The human dimension that considers the perspectives of the individual, the 
team, the management and the stakeholder. 

The SEI approach to risk assessment uses a risk taxonomy questionnaire to ensure 
that all risk areas are systematically addressed. The complete taxonomy can be reached in 
(SEI96). Table 2. 1 presents a brief summary to show the characteristics analyzed. 
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Table 2.1: SEI’s Taxonomy of Risks (SEI, 1996) 



1. Product engineering 

1.1. Requirements (stability, completeness, clarity, validity, feasibility, precedent, and scale). 

1.2. Design (functionality, interfaces, performance, testability, hardware constraints, and 
non-developmental software). 

1.3. Code and unit test (feasibility, testing, coding/implementation). 

1.4. Integration and test (environment, product, system). 

1.5. Engineering specialties (maintainability, reliability, safety, security, human factors, and 

specifications). 

2. Development environment 

2.1. Development process (formality, suitability, process control, familiarity, and product 

control). 

2.2. Development system (capacity, suitability, usability, familiarity, reliability, system 
support, and deliverability). 

2.3. Management process (planning, project organization, management experience, program 

interfaces). 

2.4. Management methods (monitoring, personnel management, quality assurance, and 
configuration management). 

2.5. Work environment (quality attitude, cooperation, communication, and morale). 

3. Program constraints 

3. 1. Resources (schedule, staff, budget, and facilities). 

3.2. Contract (type of contract, restrictions, and dependencies). 

3.3. Program interfaces (customer, associate contractors, subcontractors, prime contractor, 

corporate management, vendors, and politics). 



The SEI approach however presents the following problems: 

(a) Many of the items covered by this taxonomy are highly subjective 
and difficult to express in terms of equations. How to measure 
politics? How to measure with confidence the morale? The only 
way is to use qualitative measures that have inherent subjectivity. 

(b) Many of the items are covered more than once, for instance, 
human factors, work environment, and budget seem to be highly 
related. 

(c) The guidelines are sets of heuristics and good practices which 
impact the success of the project and depend on human experience. 



Consequently, this approach relies on the ability of the project manager using the 
checklist. An expert is required to assess the risk. 
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2 . 



Hall 



Elaine Hall’s method for managing risk (Hall, 1997) is derived from the SEI 
model. In her opinion, four major critical success factors are responsible for risk 
management: People, Process, Infrastructure, and Implementation (P 2 I 2 ). 

(a) People participate in risk management by implementing the processes 
according to the plans, by detecting problems, communicating issues and 
introducing uncertainties in their work. People at all levels need to be 
trained, involved, and motivated in risk management. 

(b) Process must transform uncertainties into risks. The transformation is 
based on identifying the sources of risk, analyzing the risk based on some 
established criteria, planning alternative strategies for risk resolution, 
tracking the risk metrics, and resolving the risk triggering action plans. 
Unfortunately, how to perform this transformation (that is the key 
problem), is not addressed in (Hall, 1997) and neither in (SEI, 1996). 

(c) Infrastructure establishes the culture that supports risk management. 

(d) Implementation is the execution of the plans, assigning responsibilities, 
authorities, tools and methods. 

In Hall’s method, checklists based on SEI taxonomy, work breakdown 
decomposition, meetings, reviews and surveys are the tools for risk identification. All 
these tools are human dependant and highly unstructured. Hence, the method is very 
difficult to automate. However, Hall emphasizes the use of metrics to identify occurrence 
of risks, such as progress in milestones, size (LOC), change (requirements added, 
changed, deleted), quality (number of defects), staff (turnover) and risk exposure. Risk 
analysis, risk planning, risk tracking and risk resolution are based on planning, and on a 
set of resolution techniques and tools inherited from SEI’s model. Hall’s approach has the 
same problems as SEI’s model. 
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3. 



Charette 



Charette introduced the concept of risk management in maintenance (Charette, 
1997). The author states that during maintenance, risk management is more difficult than 
during development. First, maintenance projects provide more opportunities for risk and 
less freedom to mitigate risk as a consequence of the previous version of the system. 
Second, risk management in maintenance requires more attention to customer-related 
issues. The approach is based on SETs taxonomy as the tool to identify treats and SETs 
software risk evaluation process to assess the risk. Charette’s approach has the same 
problems that were previously addressed about SEI’s model. The method relies on human 
experience. 



4. Jones 

During the 60’s and the 70’ s EBM has focused significantly on software processes. 
Many technologies were invented in IBM’s laboratories: HIPO diagrams, joint 
application design, formal inspections, structured walkthroughs, integrated cost and 
estimation tools, and formal specifications. Significantly, CMM has characteristics that 
can be traced back to IBM when Watts Humphrey was at IBM. Neither SEI’s CMM or 
Software Productivity Research (SPR) (Jones, 1994) addresses how to solve the problems 
of estimation. SPR is a software process introduced by Capers Jones that has some 
characteristics very similar to CMM. Jones and Humphrey were working at IBM during 
the seventies, so it is not surprising that both models have common characteristics. As an 
example the five-level scale of CMM corresponds to the five-scale of SPR. (Jones, 1994) 
observed those significant risks are not the same across all software domains. He 
introduced six categories of software projects with different kinds of risks. Table 2.2 
shows the percentage of projects at risk for each category. Note that the table is ordered 
showing on the top the risk factors more common for all the projects categories. Jones 
stated that the ten most serious risk factors observed in the SPR assessments are: 

(1) Inaccurate metrics. The generalized use of LOC as a productivity metric 
introduces errors because the differences in the languages and 
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programming styles. Counting LOC does not address the complexity 
involved in recursion nor object-oriented paradigm. LOC is very difficult 
to estimate during the requirements. Albrecht addressed this problem with 
the introduction of function points. However, recently Kitchenham, 
Kemerer and others have criticized this metric. This issue is discussed in 
Chapter V. 

(2) Inadequate measurement. Data collection is not always correctly done, 
even in the case of cost collection. One major leak in terms of cost is the 
work of end users. 

(3) Time pressure introduced by irrational schedules or by continuously 
changing requirements. This second factor is more intense as the 
complexity of the systems grows. Projects with more than 1000 function 
points are most likely to experience this problem. 

(4) Management weaknesses due to lack of education in estimation, planning, 
measuring and assessment. 

(5) Inaccuracies in cost estimation. Despite the numerous commercial 
software tools available, the use of estimation tools is not generalized. 

(6) Naive belief that moving to a new technology will create improvements in 
productivity or quality. 

(7) Late requirements. Even with the availability methodologies like 
prototyping, JAD or QFD, and metrics like function points or feature 
points, which permit to understand the impact of changes, late 
requirements continue to be a major threat. 

(8) Low quality. The current average of defects per function point in U.S. is 5 
defects per function point. 

(9) Low productivity. The current U.S. average for military projects is about 3 
function points per man-month. For MIS the productivity is about 8 
function points per man-month. 

(10) Cancellation of projects is directly proportional to their size. This 
particularly critical above 10,000 function points or 1 million LOC. 
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Table 2.2: Jones’ Top Risk Factors (Jones, 1994) 




Jones reveals some common threats characteristics of different types of software 
projects. The impact of paperwork and low productivity in DoD projects is particularly 
significant. The caveat of this work is that it does not provide a method to manage risk 
because it relies on the experience of the project manager to make the right decisions. 
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5. Karolak 

Karolak introduced a classification scheme that divides the risk in three software- 
risk elements: Technical, Cost and Schedule (Karolak, 1996). This model uses a 
subjective Bayesian probability approach to assess software risks. Each of the three 
software risk elements are influenced by ten risk factors listed in Table 2.3: 



Table 2.3: Karolak’s Scheme (Karolak, 1996) 





Software 


Risk Elements 


Software 


Technical 


Cost 


Schedule 


Risk Factor 








Organization (a) 


“low' 


HIGH 


HIGH 


Estimation (b) 


LOW 


HIGH 


HIGH 


Monitoring (cj 


“indium 


high" 


“HIGH 


Methodology (d) 


MEDIUM 


HIGH 


HIGH 


Tools (e) 


MEDIUM ' 


MEDIUM 


MEDIUM 


Risk culture (f) 


HIGH 


MEDIUM 


MEDIUM 


Usability (g) 


HIGH 


LOW 


LOW' 


Correctness (h) 


HIGH 


LOW 


LOW 


Reliability (ij 


HIGH 


LOW 


LOW 


Personnel (j) 


HIGH 


HIGH 


HIGH' 



(a) “Organization" addresses risks associated with the maturity of the organization structure, 
functions, management and communications. 

(b) "Estimation" addresses the risks associated with inaccuracies in estimating resources, 
schedules and costs. 

(c) "Monitoring" refers to risks associated with identifying problems. 

(d) "Methodology" addresses the risks associated with the lack of formal methodology and 
standards. 

(e) "Tools" refers to the risks associated with the development tools. 

(f) "Risk culture" addresses the characteristics of the management decision-making style. 

(g) "Usability" refers to risks associated to the software product after it is delivered. 

(h) "Correctness" addresses to the risks associated with compliance with requirements after the 
delivery. 

(i) "Reliability" refers to the risks of failures after the delivery. 

(j) "Personnel” includes the risks associated with the knowledge and skills of the development 
team. 



The key element to identify and measure risks on Karolak’s approach is a 
questionnaire used to evaluate the risk factors (81 questions: organization 8, estimation 7, 
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monitoring 7, methodology 7, tools 9, risk culture 11, usability 6, correctness 9, reliability 
12, and personnel 12). The answer for each question in a number between 0 and 1, where 
0 represents none and 1 represents all. The main contribution of this model is that it can 
be automated; indeed Karolak developed a tool called SERIM (Software Engineering 
Risk Model). However, the problem with this approach is that even though the tool 
provides support, human experience is still required as the key factor to identify risks. 



6. Project Management Institute (PMI) 

The Project Management Institute (PMI) introduced a methodology for risk 
management (Wideman, 1992) generalized for any kind of projects. The method is based 
on four phases: risk identification, risk assessment, risk response, and documentation. 
Risk identification follows an informal approach based on taxonomies, expert’s opinions 
and workgroup techniques. The assessment phase may range from subjective evaluation 
to the use of metrics. This phase also includes the analysis of impact. In this model there 
are two planning activities: response planning, and contingency planning; and three 
typical risk response strategies: avoidance, deflection, and absorption. PMI uses the term 
"risk" to denote two different concepts: the probability of occurrence of a threat and the 
threat itself. Another terminology issue in this approach is the use of the term "risk" in 
scenarios in which decisions are made under uncertainty rather than risk. The approach is 
too general to be useful in software engineering. 



7. Mitre Corporation 

The Mitre Corporation developed a Web application (RAMP) to capture risk 
management experience and retrieve experiences from other projects and advice. The user 
introduces the characteristics of his project in a static HTML form. A query is launched 
over the RAMP databases creating a dynamic HTML form with a set of projects with 
similar characteristics. The user can select one or more of these projects and a second 
script retrieves risks from the database. The result of this second query is a report 



35 



containing links to the applicable documents (Garvey, 1997). This approach helps the 
decision-maker by providing him related documents about similar projects, but it does not 
release the need of human experience to manage risk. 



8. Rockwell 

At Rockwell, an improvement on communicating risks more effectively provided 
the following benefits: predictable program performance, better reviews, improved 
process, and improved management practices. Three key elements are the reason for 
successful risk management at Rockwell: repeatable process, widespread access to 
adequate knowledge and functional behavior (defined as human factors). 

Functional behavior implies human interactions, motivations and incentives, 
perceptions and perspectives, communication and consensus, and decision making and 
risk tolerance. (Gemmer, 1997) identified the following functional behaviors: 

• manage risk as an asset 

• treat decision making as a skill 

• actively seek risk information 

• seek diversity in perspectives and information sources 

• minimize uncertainty on time, control and information 

• recognize and minimize bias in perceiving risk 

• plan for multiple futures 

• be proactive 

• improve the decision-making skills 

• reward those who identify and manage risks early 

Gemmer identified the following causes for risks: uncertainty in time, uncertainty 
in control, and uncertainty in information. Risk management is usually an uncertainty 
scenario characterized by: a) uncertainty in the impact or consequence, b) a time frame to 
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prevent or mitigate risks exists, c) a coupling or domino effect exists, d) uncertainty about 
the probability distribution function exists (Gemmer, 1997). 



9. Boehm 

Boehm has been studying the problem of risk management for more than one 
decade. His contributions to the area are notable. He introduced the importance of 
verification and validation of software requirements and design specifications during the 
early phases of the project as a way to mitigate risk (Boehm, 1984). Such activities 
include: completeness, consistency, feasibility, and testability of the specifications. 
Completeness implies that all the documents and references exist and that there are no 
missing items, functions or products. Consistency is both internal and external, and 
implies traceability. Feasibility requires validation that the project can be achieved with 
the actual resources, that it will satisfy the users’ needs, that it will be maintainable, and 
that its risk has been estimated. Testability requires unambiguous and quantitative 
specifications. 

Boehm introduced the Spiral Model (Boehm, 1988) as a substitute to the Royce’s 
Waterfall model. The Spiral model was the first software process in which risk 
assessment was the driven factor. The author recognized however that numerous 
difficulties in applying his model exist: 

• matching the evolving process with contracts 

• relying on risk-assessment expertise, the model is people dependent in terms 
of identification, management and risk-driven specification 

• the need of further elaboration in the spiral steps (Boehm88) 

• ambiguities about how to initiate, terminate and iterate within the spiral 

• complexities in handling incremental development, such as refinements from 
previous versions 

• difficulties in formalize processes 

• some steps were more complex than were envisioned (Boehm, 1988a) 
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Boehm introduced a method for risk management that is summarized on Table 
2.4. Risk management is divided in two families of activities: risk assessment and risk 
control (Boehm, 1989) and (Boehm, 1991). 



Table 2.4: Boehm’s Classification (Boehm, 1991) 

A. Risk Assessment is decomposed into: 

(1) Risk identification by use of checklists, decision driver analysis, assumption 
analysis, and decomposition. 

a. Checklist (top 10 risks) 

• Personnel shortfalls 

• Unrealistic schedules 

• Requirement risks 

• Developing the wrong functionality 

• Developing the wrong user interface 

• Developing extra functionality not essential or with marginal 
usefulness 

• Continuous stream of requirement changes 

• Problems in external components 

• Problems in external tasks 

• Performance shortfalls. Straining computer science 
capabilities (trying to do more than the possibilities of the 
state of the art technology): distributed processing, AI, 
human-machine interface, algorithm speed and accuracy, 
computer security, reliability and fault tolerance. 

b. Decision driver analysis: 

• Politically driven decisions 

• Marketing driven decisions 

• Applying the wrong solution to the problem because there 
exist compromises or preferences 

• Short-term versus long-term decisions 

c. Assumption analysis 

• Comparison with previous experience 

• Pessimistic approach (Murphy’s Law) 

d. Decomposition 

• Pareto 80-20 phenomena 

• Task dependencies (high fan-in implies risk: if anything slips, 
the project aborts. High fan-out also implies risk: if the 
precondition slips, then the effect is in many parts of the 
project) 

• Uncertainty areas in the plan 

(2) Risk Analysis: 
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a. Decision trees 

b. Network analysis using PERT and probabilistic network analysis 

c. Cost risk analysis using COCOMO, Putnam or other estimation tool 
for effort and duration 

d. Automated analysis tools (PROMAP, PROSIM, RISNET, SLAM, 
Opera/Open Plan, PRISM, REP) 

(3) Risk Prioritization: 

a. Assess the risk probabilities from historical data, Delphi or other group 
technique 

b. Deal with compound risks 

c. Deal with triggered risks (dominoes effect). 

B. Risk control is decomposed into: 

(1) Planning 

(2) Resolution 

(3) Monitoring (milestone tracking and top- 10 risk tracking) 



Boehm warned that current approaches to the software process may have a 
tendency to create high-risk commitments. "The waterfall model tempts to over promise 
software capabilities in contractually binding requirements specifications before 
analyzing the implications. The evolutionary development makes too easy to introduce 
new ideas and requirements that can lead to a disaster." (Boehm, 1991). In an article 
coauthored with De Marco, they showed a pessimistic and pragmatic view stating "doing 
software risk management makes good sense, but talking about it can expose you to legal 
liabilities. If a software product fails, the existence of a formal risk plan that 
acknowledges the possibility of such a failure could complicate and even compromise the 
producer’s legal position." (Boehm, 1997). 

Boehm’s contributions to risk management are multiple. This research picked the 
most important ones, such as the Spiral model, his analysis of the activities required for 
risk management, and his risk management method. Due to its relevance, a separate 
section includes a discussion about COCOMO. Despite his contributions, Boehm 
recognizes that the issue of relying on humans to assess risk remains unsolved. The use of 
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checklists, decision driver analysis, assumption analysis, and decomposition is not 
enough to automate risk identification and assessment. 



10. McFarlan 

McFarlan introduced a model to assess risk on information system projects based 
on a three-dimensional checklist covering the three major dimensions which influence the 
risk inherent in a project: (McFarlan, 1974) 

• project size in terms of budget, staffing levels, elapsed time and number of 
departments affected 

• experience with the technology 

• project structure in terms of defining the tasks and deliverables 

The importance of his contribution resides in the identification of different facets 
on software projects. This model relies on checklists and in the experience of the 
decision-maker to evaluate risk. 



11. Gilb 

In his classical text on Software Engineering Management Gilb presented a set of 
principles or rules of engagement with risk (Gilb, 1988). The approach is informal. Gilb’s 
principles are heuristics and were the state of the art at that time. His work was included 
because he was a pioneer in recognizing the problem and in recognizing the need to be 
proactive. 



12. USAF 

(USAF, 1988) defines risk as the probability at a given point in a system’s life 
cycle that the predicted goals could not be achieved with the available resources. Due to 
the high degree of uncertainty, high precision is not useful during the early phases. As the 
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system progresses, the uncertainty is transformed into risk; therefore, higher precision is 
required. The US AF introduced a method to abate risk based on checklists and estimating 
the probability of occurrence and effects. They decompose the software risk in four 
dimensions: performance, support or maintainability, cost and schedule. The effects on 
the project are categorized into catastrophic, critical, marginal and negligible. The four 
risk dimensions are measured in terms of their probability of occurrence and their effect 
according to Table 2.5. 



Table 23 : USAF Scheme for Risk 



Prob. 


1.0 -0.7 


0.7 - 0.4 


0.4 - 0.0 


0.0 


Impact 


Frequent 


Probable 


Improbable 


Impossible 


Catastrophic 


HIGH 








Critical 




^ MODERATE 


NONE 


Marginal 






' k *3jj 5 . 




Negligible 






LOW 





The USAF method is very simple and robust. However, it is informal, relying on 
checklists and the experience of the evaluator. 



D. ESTIMATION MODELS 

This section presents three models to estimate effort and duration in software 
projects: COCOMO, Putnam and function points. These estimation models are important 
because they constitute a preliminary approach to assess risk. 

1. The COCOMO Family 

Constructive Cost Model (COCOMO) introduced by (Boehm, 1981) is a family of 
models constituted by Basic, Intermediate and Detailed COCOMO. Basic COCOMO is 
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an easy to calculate model applicable to small to medium software projects. Intermediate 
COCOMO is based on the Basic model and includes effort adjustment factors. The 
detailed COCOMO explains the influence of these additional factors on individual project 
phases. These earlier models are known as COCOMO 81. 

Projects are classified into three categories: 

• organic which are characterized by small size, small teams and low 
environmental noise 

• embedded characterized by strong complex coupling with hardware or 
other kind of tight constraints like real time systems 

• semidetached which are intermediate between the previous two 
categories. 

The details of the model can be found in (Boehm, 1981), but it is important to 
highlight the following assumptions that show the optimistic bias of the model. 

• The model assumes that the requirements are defined and that they will 
remain unchanged. 

• The development period according to COCOMO 81 starts at the 
beginning of the design phase. The requirements phase is not covered. 

• The estimation covers only the direct-charged labor. In other words the 
effort applied in meetings and communication is not considered. 

• The model assumes that a man-month is 152 hours of working time. 

• The model assumes that the project will have good management. 

The input parameter for COCOMO 81 is the size estimation in KLOC, which 
constitutes a drawback because of the difficulty of predicting the size during early stages. 
COCOMO II addresses the problem of size estimation introducing a more abstract 
indicator of size called object points (a variation of function points also called application 
points). Object points can be used as input for the model in the case of small projects that 
can be developed in a few months. For bigger projects the input parameter for the model 
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is an estimate of the size in lines of code. Object points can be used to derive an 
estimation of the lines of code. (Boehm et al, 2000). This model was calibrated using 83 
projects (Chulani, et al., 1999), (Boehm, 2000). 

COCOMO 81 was not designed for evolutionary software processes. There is no 
reference to the evolutionary prototyping or to the spiral model in the book (Boehm, 
1981). The reason is because the spiral model was introduced later (Boehm, 1988). 
Moreover, in the recent book Boehm states that "COCOMO 81 did not plan for 
evolution", it was "built on the 1970’s waterfall process framework" (Boehm et al, 2000 
p. 3, 4). The Intermediate and Detailed COCOMO were designed "for cost estimation in 
the more detailed stages of software product definition." (Boehm, 1981 p. 114). The 
Intermediate and Detailed models require the introduction of subjective cost estimators. 
The Detailed Model was designed to for cost estimation in base of "phase distribution 
effort." But the word "phase" here refers to the phases in the waterfall not to the cycles in 
evolutionary prototyping. 

COCOMO Intermediate and Detailed can be used for adaptations of existing 
software, but always requiring an input in terms of lines of code. The estimation of the 
adaptation size require to know: 

• The number of delivered lines of code adapted from the existing software to 
form the new product. 

• The percentage of design modified. 

• The percentage of code modified. 

• The percentage of effort required to integrate the adapted software. 

The adaptation should not be confounded with a build in the evolutionary software 
process. The adaptation is related to the maintenance. It is a construction of a new 
product based on a previous developed software following the waterfall model. 

COCOMO II can be configured for different software processes including the 
evolutionary ones. However, it requires an estimation of the size of the product at each 
evolutionary cycle (Boehm et al, 2000). 
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2. Putnam 

In the 50’s, Peter Norden from IBM developed a manpower model. He used the 
following curve of the Weibull distribution family, named after the 19 th century physicist 
Lord Rayleigh: 

y = K ( 1 - exp(-at 2 )), and its first derivative 

y’ = 2 K a t exp(-at 2 ), where 

y = cumulative percentage of total effort 

y’ = manpower rate in terms of people per unit of time 

K = effort in man-unit of time 

t = development time 

a = a constant governing the time to manpower peek. 

Putnam, an alumnus of the Naval Postgraduate School, introduced in the 70’s a 
model applying the concepts developed before by Norden at the IBM development 
laboratory of Poughkeepsie. This model is supported by a commercial tool named SLIM 
(Software Life Cycle Management). The use of the Rayleigh curve as a reasonably good 
fit for the manpower distribution has been proved by Norden, (Putnam, 1980) and 
(Boehm, 1981). Putnam observed that a strong correlation between lines of code and 
schedule, manpower and defects exists. He recognized differences in terms of 
development difficulties between real time systems and normal information systems 
(Putnam, 1980 and 1996). Putnam’s model is based on the following assumptions 
(Londeix, 1987): 

• A development project is a finite sequence of purposeful, temporally ordered 
activities, operating on an homogeneous set of problem elements, to meet a 
specified set of objectives. 

• The number of problem elements is unknown but finite. 

• Problems are detected, recognized and solved by applying effort. 

• The occurrence of problem solving follows a Poisson process. 

• The number of people working in the project is proportional to the number of 
problems to be resolve at that time. 

The main equation of this model relates the size of the project in lines of code to 
the effort and the schedule: 
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S = C k K I/3 t d 4/3 , where 

S = number of delivered source instructions 

K = life-cycle effort in man-years 

t d = development time in years 

C k = "technology constant" that requires fine-tuning 

The required development effort (DE) is estimated as 40% of the life-cycle effort. 

That is: 

DE = 0.4 K = 0.4 (S/C k ) 3 ( l/t d 4 ) 

One difficulty of the approach, as with COCOMO, is the need of knowing the 

number of lines of code at the beginning of the project. Putnam suggests using the Delphi 

method to estimate S. 

Let a = minimum size estimation, 
b = most likely size, 
c = maximum size estimation. 

The estimator of the expected size, E(S) = (a + 4b + c) / 6. 

And the estimator of the standard deviation is s = (c - a) / 6. 

Another difficulty is to estimate the technology constant C k . Putnam suggests 
deriving it from previous projects. That is, analyzing post-mortem projects with known S, 
K and t d it is possible to derive the value of C k . This approach introduces two constraints: 

(1) To apply the model, available historic data is required. 

(2) The development process must be repeatable, that is at least CMM level 2. 
(Boehm, 1981) states that this method is not good for projects employing 

incremental development, but this comment could be a little biased. Nevertheless, 
changes in requirements lead to a new estimation. According to Putnam, the method is 
not precise for small projects with development time of two years or less. This seems to 
be because of a more rectangular manpower pattern observed in small projects. The 
method has been verified with more than 4,000 projects. Conte also observed that the 
model works "reasonably well" on very large systems but overestimates the effort on 
medium and small ones (Conte, 1986). Other criticisms of the same authors point to 
exaggeration of the effects of time compression, excessive weight on the size, and 
excessive sensibility to changes of the technological constant. 
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During this research an experiment was conducted to compare Putnam’s model 
with COCOMO 81. The experiment consisted in comparing the estimates of 100 projects 
with sizes from 10KLOC to 1MLOC using Basic COCOMO for organic, semidetached 
and embedded systems with Putnam estimation. To avoid problems of tuning, the effort 
in Putnam used the average of the development times of COCOMO. Similarly, the time 
in Putnam was calculated using the average of COCOMO efforts. In both cases a constant 
of technology = 10100 as suggested in (Boehm, 1981) was used. The following graphs 
show the findings: 

(1) In terms of effort, Putnam’s model is almost the average of embedded and 
semidetached COCOMO (Fig. 2.3). 

(2) In terms of development time, the models are quite similar, Putnam’s 
estimation being more optimistic (Fig. 2.4). 
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Figure 2.3: Effort Estimated Using COCOMO and Putnam Models 



46 




Figure 2.4: Development Time Estimated Using COCOMO and 

Putnam Models 



3. Function Points 

Functional complexity has been studied for years because it correlates highly with 
effort and risk. The traditional functional complexity metric has been introduced by 
(Albrecht, 1979 and 1983). Note that functional complexity includes to notions of 
complexity. First, there is the notion of relational complexity describing the mechanistic 
view of the system. This notion can be objectively measured. Second, there is a rational 
notion of complexity that is subjective and depends on cognitive limitations of the 
observer. Function Points had an enormous success because: 

(1) It is an early metric. It can be calculated after the preliminary analysis of 
the system. 

(2) It is easy to calculate. There are only five input parameters to compute and 
fourteen fine-tuning adjustments, but the whole process can be done 
manually. 

(3) It is the first metric that related complexity to number of lines of code. 
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The procedure for calculating Function Points is quite simple. Count the number 
of inputs, outputs, queries, files, and system’s interfaces required. Each of the five 
parameters is classified into simple, medium or complex. Depending on the parameter 
and its complexity, the count is multiplied by a weight factor. Table 2.6 presents the 
template for the calculation. 



Table 2.6: Function Points Calculation (Albrecht, 1983) 
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The result of the total is called Non Adjusted Function Points. Fourteen 
adjustment factors, whose values are in the range of zero to five, describing the 
environment are added. Finally the Function Points are calculated by the formula: 

FP = NAFP * (0.65 + 0.01 * X Fj) 
where NAFP is the non adjusted Function points 
Fj is each of the fourteen adjustment factors 

Despite its attractive approach. Function Points has many weaknesses. First of all, 
the metric was derived from a study of MIS projects in the seventies. Today, there are 
many issues that are not considered by the metric and that are contributors to complexity. 
For instance, recursive functions, reuse, inheritance, communication by messages and 
polymorphism are not considered by the metric. The languages have evolved also and 
differ a lot from the COBOL of the seventies. Finally, programming styles have suffered a 
dramatic change that is not reflected in the metric. 

(Kemerer, 1993) reported some weaknesses of the metric. Similar results have 
been reported by (Kitchenham, 1993 and 1997). The main issue is that Function Points is 
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a not well-formed metric because there is a correlation between their constituent 
elements. In her conclusions she stated that: 

(1) The individual function point elements were not independent. 

(2) Not all the function point elements were related to effort. 

(3) An effort prediction metric based on inputs and outputs was just as good a 
predictor as Function Points. 

(4) An effort prediction metric based on the number of files and the number of 
outputs was only slightly worse that Function Points. 

(5) To get good estimates estimation methods and models based on the 
organization’s performance, working practices, and software experience 
were required. 

(6) Uncertainty and risk cannot be managed effectively at the individual 
project level. However, they can be managed in the organization context. 
If a single project had to be assured against all possible risks and 
uncertainty, its cost would be prohibitive. The sources for estimate 
uncertainty are the measurement error caused by model limitations and 
accuracy, the use of erroneous assumptions, and the use of the model 
outside its domain. 

Even if evidence of defects in the metric existed, nobody introduced a better 
alternative. So, Function Points remained as the most common prediction metric for many 
years. More recently, some extensions to Function Points have been introduced, such as 
“feature points” and “Boeing’s 3-F function points,” addressing the effort estimation for 
embedded systems. 



4. Conclusions about COCOMO, Putnam and Function Points 

All these methodologies have some weaknesses with respect to software 
evolution. First, the need of a size estimate as an input parameter limited the applicability 
of COCOMO and Putnam methods. Second, the characteristics counted on function 
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points are quite different from the specification attributes. Third, the criticisms introduced 
by (Kemerer, 1993) and (Kitchenham, 1993 and 1997) suggested that despite the 
correlation observed between complexity and size, other metrics could be more accurate, 
and this opened opportunities for new research. 



E. SOFTWARE RELIABILITY 

Software reliability engineering is based on a solid body of knowledge that 
includes operational profiles, random process software reliability models, statistical 
estimation and sequential sampling theory (Musa, 1995). This section describes the 
reliability models and classifies them according to the characteristics of their probabilistic 
assumptions. Software reliability is defined as the probability of failure-free software 
operation for a specified period of time in a specified environment (ANSI, 1991). 
Software reliability is related to quality. ISO 9000-3 specifies the measurement of field 
failures as the minimal required quality metric. It includes: 

a) Software reliability measurements, which includes estimation and 
prediction models. 

b) Metrics and attributes of product design, development process, system 
architecture and their relation with reliability. 

c) The application of the knowledge in specifying and guiding software 
development, system architecture, testing, acquisition, use and 
maintenance. However, the reliability approach studies the post 
implementation behavior of the software. At that stage, the product is 
already constructed. Hence, changes in the product are impossible or very 
expensive. So the "knowledge" arrives too late to be useful in the present 
project, but can be applied to future developments. (Lyu, 1995). 

Donnelly, Everett, Musa and Wilson stated that the practice of software reliability 
provides a means to "predict, estimate, and measure the rate of failure occurrences in 
software and firmware" (Lyu, 1095 pp.219). Reliability can only be reached by following 
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a rigorous development process. During the product concept development, it is necessary 
to determine the functional profile, define and classify the failures, and identify the 
customer’s reliability needs. Determining the functional profile means to specify the tasks 
to be performed and the environmental factors that could influence the process. Quality 
Function Deployment (QFD) is a useful methodology to apply. The definition and 
classification of failures must be done from the point of view of the user and represents a 
trade-off. Too many classes require excessive effort in collecting and analyzing the 
metrics. Few classes can provide too vague information. Finally, identifying the user 
reliability needs at a high level is required. A reasonable way to do this is to assess the 
reliability capabilities of similar products. 

During the requirements phase, reliability objectives need to be refined and 
specified, including customer satisfaction, performance, and trade-offs between reliability 
and other factors, such as cost, delivery time and functionality. The reliability 
requirements have strong influence in the architecture and in the future evolution steps of 
the development process. 

During the design phase, one must analyze the reliability of the components in 
order to determine the overall reliability of the system. This design phase should be 
driven by reliability objectives after the identification of risky areas. For instance, the use 
of redundant software elements could be required. 

During the implementation, one should identify the critical areas and different 
techniques that should be applied to reduce risks. Such techniques include tight 
development standards and methodology, modularity, reuse, development in evolutionary 
steps, inspections and reviews, and software configuration management. Another 
important consideration in this phase is the reliability assessment for components 
acquired or developed by outsourcing. This assessment must be done as soon as possible. 
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Part of the testing (unit test and integration) is conducted during the successive 
evolutionary steps, but in the system test many reliability issues could appear. At that 
time, the system is complete, and we can have a complete picture of its reliability prior to 
the delivery to the customer. To conduct the system test, one must determine various 
operational profiles according to the user’s point of view. The purpose is to locate stress 
points where the reliability requirements are not reached or could potentially not be 
reached. This is a particularly intensive data collection activity that requires automated 
tools. Testing could continue in beta test sites where the reliability objectives are 
certified. 

Finally, when the product is delivered to the customer, reliability should be 
monitored to assess the success of the project, to measure the quality of the process 
including the testing scenarios, and to have an indicator of customer satisfaction. The last 
is a critical success factor that needs to be tracked also by surveys and meetings, in order 
to be sure that any symptom of dissatisfaction was immediately revealed. Software 
reliability techniques are another tool available that can improve the software 
development process. Unfortunately, the models available require that the product is 
almost complete in order to predict its behavior. In the following sections we will 
describe various software reliability models classified according to the following scheme: 

a) Exponential failure time models 

b) Weibull and Gamma failure time models 

c) Infinite failures models 

d) Bayesian models 

e) Models for early stages 

1. Exponential Failure Time Models 
a. Jelinski-Moranda Model 

Jelinski and Moranda introduced this model when they were working for 
McDonnell Douglas. The elapsed time between failures has an exponential distribution 
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with a parameter that is proportional to the number of remaining faults in the software. 
Although this model has been replaced, it was important for setting the framework for 
other work in modeling. The assumptions for this model are the following: 

• The rate of fault detection is proportional to the current fault content of the 
software. 

• The fault detection rate remains constant in each interval between faults. 

• The correction of a fault is instantaneous and does not introduce new 
faults. 

• The operation of the software is similar to the conditions in which the 
prediction of reliability is done. 

• Within each severity class, the probability of finding a fault is the same. 

• The failures are independent. 

Model form 

The time between failures Xj = tj - t;.i for i = 1,. . .n 

.\ Xj are independent random variables with exponential distribution, with 
Mean failure = |i.(t) = N / (1 - exp(-<}>t)) 

Failure intensity function = A,(t) = N 0 exp(-<|)t) 

Where N = total number of faults in the software 
<J) = proportionality constant. 



b. Schneidewind Model 

This model introduced by Dr. Norman Schneidewind in (Schneidewind, 
1975) is based on the idea that the current fault rate is a better estimator of the future 
behavior than the observed rates in the distant past because the failure rates can change 
over time. So, the model weights the observations differently, according to the analyst's 
point of view. The main idea introduced by Schneidewind was to monitor the occurrence 
of software errors as a predictor for future cumulative detected and corrected errors. 
These estimations are useful in order to: 
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• Identify the trade-off function between error reduction and cost of error reduction. 

• Provide a quantitative basis for accepting or rejecting software during functional 
testing. 

• Provide a quantitative basis for deciding whether additional testing is warranted 
based on the cost of error removal. 

The model is based on non-homogeneous Poisson processes. A non- 
homogeneous Poisson process is a Poisson process in which the mean is not a constant, 
but a random variable. This model has been used extensively on IBM’s Flight Control 
software for the Space Shuttle with great success. It is one of the four selected models by 
the AIAA’s Recommended Practice for Software Reliability (AIAA, 1993) and 
considered one of the most accurate available (Lyu, 1995). Schneidewind proposed three 
forms of the model: 

Model 1: Uses all fault counts of the n periods, reflecting the view of 
equal importance. 

Model 2: Ignores the fault counts of the first s-1 periods. This reflects the 
view that the early time period contribution in predicting future behavior is insignificant. 
This is very useful to discard the confounding effect of a learning curve. 

Model 3: Uses the cumulative-fault counts from intervals 1 to s-1 as the 
first data point, and the individual counts for periods s to n, as additional data points. This 
view is an intermediate between the other two. 

The assumptions for the models are the following: 

• The cumulative number of failures by time t follows a Poisson process with mean 
|i(t) such that the expected number of fault occurrences for any time period is 
proportional to the expected number of undetected faults at that time. 

• The number of faults is finite. 

• The failure intensity function decreases exponentially with time. The failure 
intensity function A.(t) = a exp (-pt) for some a, p constants. 
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• The number of faults (fl) detected on each interval i are independent. 

• The fault correction rate is proportional to the number of faults to be 
corrected. 

• All the intervals have the same length. 

• The operation of the software is similar to the conditions in which the 
prediction of reliability is done. 

• Within each severity class, the probability of finding a fault is the same. 

• The failures are independent. 

Concerning error detection, Schneidewind’s model has the following 

assumptions: 

• The observation of the process is in discrete intervals of time. 

• The number of errors in each time interval is independent of the number of 
errors in any other interval. 

• The pdf in each interval has the same distribution but a different mean. 

• The mean of number of detected errors decreases from interval to interval 
as a result of the correction process. 

• The rate of error detection in an interval is proportional to the number of 
errors in the interval. 

Because of (2) and (3) this model is a non-homogeneous Poisson process 
with exponential decaying intensity function as stated in (4). The following equations 
summarize the detection model: 

Xj = actual number of errors during interval i 
mj = estimate number of errors during interval i 
Decaying intensity function d(i) = a exp(-pi) for a, P > 0 
Cumulative mean number of errors D(i) = (a/p) exp(-pi) 

Estimate number of errors during interval i 
mi = (a/p) {exp(-p(i - 1)) - exp(-pi)} 

Time estimated to detect D cumulative errors 
i<j = {log (o/(a-pD))}/p 

Time estimated for the detection rate to reach the value d 
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id = (log (ot/d)) / p 

Delay correction error. This is the difference in time between detection and 
correction of errors. 

C(i) = D(i - Ai) = «x/p) { 1 - exp(-p(i - Ai))} 

Time estimated to correct a cumulative number of errors 
i c = Ai + {log «X/(0C-PC))}/P 
Correction rate of errors c(i) = a exp(-P(i - Ai)) 

Time estimated to reach a correction rate c 
i c = Ai + (log (Ot/c)) / P 

Difference between detected and corrected errors 

R(i) = D(i) - C(i) = (a/P) (exp(-Pi))(exp(P Ai) - 1) for i > Ai 

Unlike hardware, which deteriorates with time, software ideally should 
improve with time. Theoretically as time goes by more errors are discovered, so the 
residual number of errors decreases. However, this is not always the case. Two factors 
contribute to reproduce errors in a software system. First, we need to consider that error 
correction is error prone. New errors could be introduced as a consequence of debugging. 
Second, and more important, software require maintenance and these major modifications 
or extensions are source of new errors. Consequently, the time series of error counts will 
not necessarily be monotonically decreasing. Our concern about the model is that the 
assumption of correction without introducing new errors seems to be optimistic. 



c. Goel-Okumoto Model 

This model uses the number of faults per unit of time as independent 
Poisson random variables. It was introduced in 1979 by Goel and Okumoto (Goel & 
Okumoto, 1979) and is the source for other models, such as the S-shaped model. The 
assumptions for this model are the following: 

• The cumulative number of failures at time t follows a Poisson process with 
mean p(t). This mean function is such that the expected number of fault 
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occurrences for any time interval (t, t+At) is proportional to the expected 
number of undetected faults at time t. It is also assumed that the total 
number of faults is finite. 

• The number of faults detected in each time interval is independent for all 
time. 

• The operation of the software is similar to the conditions in which the 
prediction of reliability is done. 

• Within each severity class, the probability of finding a fault is the same. 

• The failures are independent. 

Model form 

Mean = p.(t) = N (1 - exp(-b t)) for some constants b, N > 0 
Failure intensity function = A.(t) = N b exp(-b t) 

A variation of this model permits one to determine an optimal release time 
for a software system. This adaptation uses the time of fault occurrences instead of the 
fault counts. Given a desired reliability R for a specified operational time O, then the 
required amount that the software must be observed is: 

T = (1/b) ( ln(a(l - exp(-b O)) - (ln(ln(l/R)))) 

d. Musa’s Model 

The rationale behind this model is that the execution time is more 
reflective of the actual stress applied to software than the calendar time. Dr. John Musa of 
AT&T Bell Laboratories (Musa, 1975) introduced the model in 1975. The assumptions of 
the model are the following: 

• The cumulative number of failures by time t (M(t)), follows a Poisson 
process with mean p.(t) = p o (1 - exp(-Pit)), where P 0 , Pi > 0. The expected 
number of failures in any period is proportional to the expected number of 
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undetected failures at that time. The total number of faults that will be 
detected when time ~ is J3 0 - 

• The execution times between failures are exponentially distributed. 

• The personnel resources remain constant over the period of time the 
system is observed. 

• The system assumes a relationship between MTTF and resource 
expenditures. 

• Testing personnel can be fully utilized and computer utilization is 
constant. 

• Fault-correction personnel are assigned randomly to serve a fault queue. 
Fault correction is assumed to be a Poisson process. 

• The operation of the software is similar to the conditions in which the 
prediction of reliability is done. 

• Within each severity class, the probability of finding a fault is the same. 

• The failures are independent. 

e. Hyperexponential Model 

Ohba in 84 (Ohba, 1984) introduced the first hyperexponential model. The 
main idea of this model, which has many variations, is that a system has different classes 
of software. The software has an exponential failure rate, but each class or section has 
different rates reflecting its nature (differences in complexity, differences between 
developers, or differences in programming languages). When there are only two classes 
(e.g. old software versus new software, or easy to test versus difficult to test, etc.), this 
model is called a modified exponential software-reliability-growth model. The model has 
the following assumptions: 

• The software is composed by K sections (or classes of code) so that within 
each class: 

■ The rate of fault detection is proportional to the current fault 
content. 

■ Fault detection is constant over the intervals between faults. 
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■ Each fault is corrected instantaneously without introducing new 
faults. 

• The software as a whole has a cumulative number of failures by time t 
(M(t)), that follows a Poisson process. 

• The operation of the software is similar to the conditions in which the 
prediction of reliability is done. 

• Within each severity class, the probability of finding a fault is the same. 

• The failures are independent. 

2. Weibull and Gamma Failure Time Models 

a. Weibull Model 

This model assumes that the fault distribution has a Weibull distribution. 
Many hardware failure models are modeled with this distribution (Lyu, 1995). The main 
characteristic of the distribution is that its parameters permit great flexibility, adapting the 
model to increasing, decreasing or constant failure rates. The assumptions of the model 
are the following: 

• There is a fixed number of faults at the beginning of the experiment. 

• Each fault has a time to failure (Ta) distributed as a Weibull distribution 
with parameters a, (3: 

Ta = apt®' 1 exp(-|3t a ), with a, (3 > 0 and t > 0. 

• The number of faults detected in each time interval is independent. 

• The operation of the software is similar to the conditions in which the 
prediction of reliability is done. 

• Within each severity class, the probability of finding a fault is the same. 

• The failures are independent. 

b. S-shaped Reliability Growth Model 

The per-fault failure distribution of the S-shaped model is Gamma (Lyu, 
1995). The number of failures per period of time follows a Poisson process. This model 
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assumes a finite number of failures. The S-shape growth curve describe the testing 
process with an initial learning curve at the beginning, then a growth, and then a level at 
which errors are more difficult to detect. The S-shaped model has the following 
assumptions: 

• The cumulative number of failures by time t, follows a Poisson process 
with mean = p.(t) = a(l - (1 + Pt)e~Pt) for a, p > 0. At the limit, when 
t |i.(t) = a < =o. 

• The time between failures of the (i - l)st and the ith depends on the 
time to failure of the (i - l)st. 

• When a failure occurs, the fault is immediately removed without 
introducing new faults. 

• The operation of the software is similar to the conditions in which the 
prediction of reliability is done. 

• Within each severity class, the probability of finding a fault is the 
same. 

• The failures are independent. 



3. Infinite Failure Models 
a. Duane’s Model 

This model was originally proposed for hardware reliability (Lyu, 1995). 
This is a non-homogeneous Poisson process in which the failure intensity function has the 
same form as the hazard rate of the Weibull distribution. This model as been referred to 
as "the power model." The assumptions of the model are the following: 

• The cumulative number of failures by time t, follows a Poisson process 
with mean = p(t) = at** for a, P > 0 (If p = 1, then we have the 
homogeneous Poisson process). 

• The operation of the software is similar to the conditions in which the 
prediction of reliability is done. 



60 



Within each severity class, the probability of finding a fault is the same. 
The failures are independent. 






b. Geometric Model 

The geometric model (Lyu, 1995) is a variation of the Jelinski-Moranda 
Model. Here the time between failures is assumed to be exponential distributed with 
mean decreasing geometrically. The geometric decay reflects the smaller impact of the 
later-occurring faults. The assumptions of the model are the following: 

• The fault detection rate is a geometric progression, and it is constant 
between fault detections. 

• There are an infinite number of total faults in the system. 

• The time between detection follows an exponential distribution. 

• The operation of the software is similar to the conditions in which the 
prediction of reliability is done. 

• Within each severity class, the probability of finding a fault is the same. 

• The failures are independent. 



c. Musa-Okumoto Logarithmic Model 

The Musa-Okumoto logarithmic Poisson Model (Lyu, 1995) is another 
example of nonhomogeneous Poisson processes with an exponential decrease. The 
decrease reflects the view that the earlier discovery of failures has a greater impact on 
reducing the failure intensity function than those encountered later. The assumptions of 
the model are the following: 

• The failure intensity decreases exponentially with the expected number of 
failures experienced. 

• The cumulative number of failures follows a Poisson process. 

• The operation of the software is similar to the conditions in which the 
prediction of reliability is done. 
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Within each severity class, the probability of finding a fault is the same. 
The failures are independent. 









4. Bayesian Models 

The Bayesian approach introduces a subjective viewpoint considering that if no 
failures occur while the software is observed, then the reliability should increase, 
reflecting the growth in user’s confidence. The reliability is considered a reflection of the 
number of faults discovered and the time between failures. 

The Bayesian approach also reflects the viewpoint that different faults have 
different impacts on the reliability of the system, and the number of faults is not as 
important as their impact. If many faults exist but they seldom appear, then by using the 
Bayesian approach the system is relatively reliable. The mean time to failure is therefore a 
very important metric in this framework. 



a. Littlewood-Verrall Reliability Growth Model 
This model tries to account for the fault generation during the fault 
correction process. This approach is very realistic because the software could become less 
reliable than before. Because of the uncertainty, each new version can be better or worse 
in terms of reliability. The distribution of failure times is random. The assumptions of this 
model are the following (Lyu, 1995): 

• Successive execution times between failures are assumed to be 
independent exponential random variables with parameter i = 1,..., n. 

• The ^j's form a sequence of independent random variables, each with 
gamma distribution. 

• The software is operated in the specified or normal way. 
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b. Other Bayesian models 

Other Bayesian models include variations of several models, such as 
Jelinski-Moranda, Jewell, Littlewood and Sofer, Kyparisi-Singpurwalla, Lyu, Becker- 
Camarinopoulos, and Thompson-Chelson (Lyu, 1995). 



5. Software Reliability Prediction in Early Stages 

The prediction of software reliability during testing is useful to understand the 
future behavior of the software in the operation phase. However, this benefit is limited 
because it is too late to incorporate changes without incurring high costs and schedule 
overruns. If a prediction could be made in the early phases, where changes can be 
introduced without excessive costs and without a serious impact on the schedule, then a 
drastic improvement could be reached. Moreover, this prediction could be a great 
advance in software engineering. Few models have been addressed to predict software 
reliability during the early phases of the project. 



a. Phase-Based Model 

Gaffney and Davis in (Gaffney, 1988) introduced the Phase-based model 
using statistics obtained during reviews requirements, design and implementation. The 
model is based on the following assumptions: 

• The staffing level of the development effort is directly proportional to the 
number of faults discovered during each phase. 

• The fault discovery curve is monomodal. 

• A good estimation of code size exists. 

The number of discovered faults per line of code from phase t-1 to phase t 

is given by: 

AVt = E ( exp(-B(t-l) 2 ) - exp(-Bt 2 )) 

where E = the total lifetime fault rate is expressed in terms of KLOC 
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t = the discovering index associated with the phase (requirements: t = 1, 
design: t = 2, implementation: t = 3, unit test: t = 4, software integration: t 
= 5, system test: t = 6, acceptance test: t = 7). 

B = a defect discovery constant. 

A fourth assumption implicit in the model is that the development process 
follows the waterfall life cycle. This is an important restriction for adapting the model to 
the new evolution development processes. 



b. Agresti-Evanco Ada Model 

Agresti and Evanco introduced in (Agresti, 1992) a prediction model that 
addresses the particularities of Ada language. The model is a multivariate linear 
regression with dependant variable the log of the default density. Among the independent 
variables are architectural complexity of design, volatility, reuse and functional 
complexity. This model seems to have an interesting level of accuracy. According to 
(Lyu, 1995), 63 to 74% of the variation can be explained with this model. However, the 
applicability to Ada language projects restricts the model considerably. 



c. Rome Lab Model 

The US Air Force’s Rome Laboratory introduced this model in 1992 
(Rome Lab, 1992) to predict the initial fault density as a function of the following factors: 

• Application type, differentiating projects in three categories: real time, 
scientific and managerial. 

• Development environment, considering the differences in tools and 
methodologies according to three categories: organic, semidetached and 
embedded. 

• Requirements and Design metrics: anomaly management, tracebility, and 
existence of software quality assurance in the process. 
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Software implementation metrics: development language level, size, 
modularity, reuse, complexity, and the existence of review standards. 






6. Conclusions about Software Reliability Models 

The main contribution of this set of research is the emphasis on solid statistical 
foundations to assess reliability. Some of the distributions used, such as non- 
homogeneous Poisson processes and Weibull, are interesting to model life cycles. The 
caveat of this approach is that the conclusions from the use of those models arrive too late 
in the life cycle to provide effective support from the engineering point of view. 



F. MODERN PROJECT MANAGEMENT TECHNQUES: ViteProject 
1. ViteProject 

ViteProject is a modeling and simulation tool that integrates the organizational 
work of projects explicating the interdependencies between tasks and roles not only from 
the point of view of producer-consumer, such as in CPM or Pert, but also communication 
and rework dependencies. ViteProject is the commercial version of VDT (Virtual Design 
Tool), a research based on contingency theory directed by Dr. Raymond Levitt at Stanford 
(Jin, 1996). CPM models are sequential interdependencies through explicit representation 
of precedence relationships between activities. This simplified vision of the project 
cannot address the dynamics created by reciprocal requirements of information in 
concurrent activities, exception management, and the impacts of actor interactions. This 
issue is addressed by VDT. The original model of VDT was based on the following 
observations about collaborative, multidisciplinary work in large complex projects: 

• Organizational tasks in the project can be divided into two categories: 
production work that directly adds value to the product, and coordination work 
that facilitates the previous one. 
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• Contingency theory provides qualitative insights about the extent of 
coordination work, but did not provide information about how to address the 
bottleneck problems created by coordination. 

The model integrates the micro-level description of the entities that perform work 
and process information called "actors." Actors can be individuals or small teams 
working as a unique and cohesive unit where individuals are not differentiated. Actors 
have two basic behaviors: attention allocation and information processing. As a 
consequence of such behaviors, actors perform production and coordination. The model 
is based on the following assumptions: 

• Actor allocation assumption: Each actor has one input buffer where all the 
incoming information and requests for production or coordination work arrive. 
The input buffer is a queue that supports different policies: priorities, FIFO 
and random. Each actor also has an output buffer to place its accomplished 
work. 

• Actor capacity allocation assumption: An actor has a certain information- 
processing capacity determined by its skill type, skill level, and allocable time. 
An information processing work can be processed and completed if the actor 
allocates sufficient capacity to the job. This assumption implies: a) 
information processing requires not only attention but also takes time; b) the 
information content of a work is related to the skills; c) the volume of a work 
is related to the time; d) actors have limited capacity to allocate. 

• Actors cannot allocate 100% of their capacity to work because they are 
interrupted by: a) information requests from other actors; b) decision-making 
to solve exceptions produced by subordinate actors; c) meetings; and d) 
processing noise, that is all other interruptions created outside the project that 
impact the actor. 

The organization structure is modeled through simulation. The organization 
variables, such as control structure, communication structure, formalization and matrix 
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strength, influence the actor’s micro level actions, and consequently an organization’s 
emergent performance appears. 

In a Vite project, an activity is any work that consumes time and may generate 
communications or exceptions. Each activity has a set of properties that include: name, 
description, work (time), a number of sub-activities, priority, skill required, and QFD 
analysis measures (requirement complexity, solution complexity, and uncertainty) (Levitt, 
1999). For each activity in the project, ViteProject requires an estimation of its 
complexity in terms of its effort or duration. This is expressed in FTE (full time 
equivalent). One person working full-time is one FTE (Levitt, 1999). The work required 
for the activity depends on its complexity, and is estimated as the total time (including 
time of all necessary workers) required to do effective work on the activity. This does not 
include ineffective work, such as rework, waiting, or coordination. This number is 
sometimes referred to as the “work volume” or the “effort hours” for the activity. The 
activity duration is computed as: duration = (work volume)/(FTE assigned). 

Complementarily, Vite provides adjustments to refine the complexity of each 
activity, and to express the difficulty of the elucidation of the requirements. ViteProject 
provides three parameters to adjust the complexity and the uncertainty of the 
requirements (Levitt, 1999) 5 : 

1. Solution Complexity refers to the extent to which an activity's solution is 
affected by other activities. 

2. Requirements Complexity models the number and difficulty of the functional 
requirements that need to be satisfied to complete the activity. 

3. Uncertainty represents the extent to which information needed to complete an 
activity is unavailable at the time the activity starts. The missing information 
could be the output of a concurrent activity, the missing information about a 
client requirement, or the missing information about an unknown state of 
nature. 



5 



The details of the configuration of ViteProject are presented on Chapter V, Section C.6 and Table 5.1. 
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